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METHOD FOR PRODUCING ENHANCED-RESOLUTION 
IMAGE BY USE OF A PLURALITY OF LOW- 
RESOLUTION IMAGES 

BACKGROUND OF THE INVENTION 

5 1. Field of the invention 

This present invention relates to an image processing method, and more 
particularly to a method for producing an enhanced-resolution image by use of a 
plurality of low-resolution images. With respect to the technology background of the 
method of the invention, please refer to the following references: 

10 [1] M. Irani and S. Peleg, "Improving Resolution by Image Registration," 

CVGIP: Graphical Models and Image /Voc.,1991, Vol. 53, pp. 231-239; 

[2] R. Y. Tsai and T. S. Huang, "Multiframe Image Restoration and 
Registration," in Advances in Computer Vision and Image Processing, Vol. 1 (T. S. 
Huang, ed.), Greenwich, CT: Jai Press, 1984, pp. 317-339; 

15 [3] P. Cheeseman, B. Kanefsky, R. Kruft, J. Stutz, and R. Hanson, "Super- 

Resolved Surface Reconstruction from Multiple Images," NASA Technical Report 
FIA-94-12, 1994; 

[4] A. M. Tekalp, M. K. Ozkan, and M. I. Sezan, "High-Resolution Image 
Reconstruction for Lower-Resolution Image Sequences and Space-Varying Image 
20 Restoration," IEEE International Conference on Acoustics, Speech, and Signal 
Processing, San Francisco, CA, 1992, Vol. Ill, pp. 169-172; 

[5] R. C. Gonzalez and R. E. Woods, Digital Image Processing, Addison- 
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Wesley, Reading, MA, 1992; 



[6] W. K. Pratt, Digital Image Processing, 2nd Ed., Wiley, New York, 2001; 
[7] U.S. patent NO. 6,330,344; and 
[8] U.S. patent NO. 5,969,848. 

2. Description of the prior art 

Due to environmental constraints and resolution of image sensors, sometimes 
we can only get low-resolution images. In order to improve the image quality and 
resolution seen by human eyes, more than one input image is required. With image 
sequences, a blurring scene, a dim figure, or an unclear object of poor quality can be 
reconstructed to an enhanced-resolution output image and can then be easily 
observed and recognized. 

Prior researches regarding the reconstruction of an enhanced-resolution image 
by use of low-resolution images are mainly divided into iterative methods [1], 
frequency domain methods [2], and Bayesian statistical methods [3]. In aforesaid 
methods, so far, the iterative algorithm that has been developed by Irani [1] in 1991 
and reconstructs an enhanced-resolution image mainly by image registration, is still 
most reliable in the prior arts regarding the reconstruction of an enhanced-resolution 
image. The iterative method mainly consists of three phases: initial guess, imaging 
process, and reconstruction process. The procedures of the three phases of the 
iterative method will be described in details in the detailed description of the 
invention. 

However, it is noticed that the iterative method will consume more computation 
time as the magnification factor predetermined in the iterative method becomes 
larger, i.e., the size of the reconstructed image becomes larger. Typically, the 



runtime of image reconstruction by the iterative method is on the order of hours and 
depends on the performance of computer system. 

Therefore, an objective of the invention is to provide a method for 
reconstructing an enhanced-resolution image with improved enhanced-resolution 
5 algorithms, which is based on Irani' s iterative method and employs well-suggested 
initial interpolation, automatic image selection and robust image registration. Further, 
the enhanced-resolution image reconstructed by the method of the invention can be 
performed by an image enhancement post-process to enhance image quality thereof. 

Whereas the conventional systems of the reconstruction of enhanced-resolution 
10 images work well only as the low-resolution image sequences are captured by 
moving a stationary camera in a constant displacement in relation to the whole scene, 
i.e., the targets, needed to be reconstructed, associate with stationary scenes. 
Therefore, another objective of the invention is to provide a method for 
reconstructing an enhanced-resolution image, which work well not only for the 
1 5 conditions as the targets, needed to be reconstructed, associate with stationary scenes, 
but also for the conditions as the targets, needed to be reconstructed, associate with 
moving objects. 

SUMMARY OF THE INVENTION 

An objective of the invention is to provide a method for reconstructing an 
20 enhanced-resolution image with improved enhanced-resolution algorithms. 
Moreover, the method of the invention, based on the iterative method, utilizes well- 
suggested initial interpolation, automatic image selection and robust image 
registration. Further, the enhanced-resolution image reconstructed by the method of 
the invention can be performed by an image enhancement post-process to enhance 
25 image quality thereof. 

Another objective of the invention is to provide a method for reconstructing an 
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enhanced-resolution image, which work well not only for the conditions as the 
targets, needed to be reconstructed, associate with stationary scenes, but also for the 
conditions as the targets, needed to be reconstructed, associate with moving objects. 

According to a preferred embodiment of the invention, M first low-resolution 
5 images are used to reconstruct an enhanced-resolution image. The M first low- 
resolution images associate with a scene. Each of the M first images is represented 
by a set of uniformly sampled and quantized pixels. A relationship of a 
magnification factor exists between the enhanced-resolution image and each of the M 
first images. First, the M first images are stored. Then, one is selected from the M 

10 first images as a prototype image, and the non-selected (M-l) first images are 
referred to being as (M-l) second images. Based on the magnification factor, extra 
pixels are interpolated into the set of pixels of the prototype image, and then each 
value of extra pixels is calculated according to the values of neighbor pixels thereof 
to produce an interpolated prototype image. The respective translations, which each 

15 exists between each of the (M-l) second images and the interpolated prototype image, 
are also calculated. The translations of the (M-l) second images are divided by the 
magnification factor, respectively, to obtain the modulus with respect to the 
translations of the (M-l) second images. Based on a criteria, one is selected from the 
second images whose related modulus are the same. The selected second images 

20 together with the rest of the second images, whose related modulus are not the same, 
are referred to being as N third images where N is equal to or less than (M-l). The 
interpolated prototype image is down-sampled N times according to the respective 
translations between each of the N third images and the interpolated prototype image 
to produce N fourth images which each corresponds to one of the N third images. 

25 The difference between each of the N third images and the corresponding fourth 
image thereof is calculated. The values of the pixels of the interpolated prototype 
image are adjusted according to an average of the differences between each of the N 
third images and the corresponding fourth image thereof. The steps of down- 
sampling and of adjusting the values of the pixels of the interpolated prototype image 
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are repeated until the values of the pixels of the interpolated prototype image 
converge to a satisfactory result. Finally, the interpolated prototype image whose 
values of pixels converge to the satisfactory result is referred to being as the 
enhanced-resolution image. 

5 The advantage and spirit of the invention may be understood by the following 

recitations together with the appended drawings. 

BRIEF DESCRIPTION OF THE APPENDED DRAWINGS 

FIG. 1 is a flowchart showing workflow of the method according to the 
invention. 

10 FIG. 2 A and FIG. 2B are schematic diagrams showing the calculation of P value 

by use of a third-order interpolation way. 

FIG. 3 evaluates the performance of interpolation ways of different orders by 
Peak Signal-to-Noise Ratio (PSNR) between the original image and reconstructed 
images. 

15 FIG. 4A shows schematically the image registration using local matching 

technique. 

FIG. 4B shows schematically the image registration using global matching 
technique. 

FIG. 5A through FIG. 5D show the results about the first case of reconstructing 
20 enhanced-resolution image according to the method of the invention. 

FIG. 6A through FIG. 6D show the results about the second case of 
reconstructing enhanced-resolution image according to the method of the invention. 
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FIG. 7 shows PSNR regarding the second case as the number of iterations grows. | 

FIG. 8A through FIG. 8D show the results about the third case of reconstructing 
enhanced-resolution image according to the method of the invention. 

DETAILED DESCRIPTION OF THE INVENTION 

5 The invention is to provide a method for reconstructing an enhanced-resolution | 

image with improved enhanced-resolution algorithms, which is based on Irani 's 
iterative method and employs well-suggested initial interpolation, automatic image 
selection and robust image registration. The enhanced-resolution image, i 
reconstructed by the method of the invention, can further be performed by an image f 

1 0 enhancement post-process to enhance image quality thereof. Hereinafter, the steps of 
the method according to the invention and preferred embodiments of the invention 
will be described in details to point out the spirit, characteristics and advantages of 
the invention. Those will also be illustrated as follows how to reconstruct an 
enhanced-resolution image associated with a stationary scene and how to reconstruct 

1 5 an enhanced-resolution image associated with a moving object. 

Referring to FIG. 1, FIG. 1 is a flowchart showing the workflow of the method 
according to the present invention. It is noted that M first low-resolution images are 
used to produce an enhanced-resolution image in the method according to the 
invention. Moreover, the M first low-resolution images associate with the same 
20 scene, i.e., the M first low-resolution images all are captured from the same scene. 
Each of the M first images is represented by a set of uniformly sampled and 
quantized pixels. A relationship of a user-desired magnification factor exists 
between the enhanced-resolution image and each of the M first images. 

At the start, the method, according to the invention, performs step S10 to store 
25 the M first images. 

i 



6 



The method of the invention then performs step S12 to select one of the M first 
images as a prototype image, and to refer the non-selected (M-l) first images to being 
as (M-l) second images. 

Subsequently, the method of the invention performs step SI 4. The initial guess 
5 phase of the method of the invention is achieved during step SI 4. Step S14 is to 
interpolate extra pixels into the prototype image based on the magnification factor. 
In step SI 4, the values of the extra pixels are calculated according to the values of 
neighbor pixels thereof to produce an interpolated prototype image, and the 
respective translations which each exists between each of the (M-l) second images 
10 and the interpolated prototype image are also calculated. 

After step SI 4, step SI 6 is performed. The automatic image selection phase of 
the method of the invention is achieved during step SI 6. Step SI 6 is to divide the 
translations of the (M-l) second images by the magnification factor, respectively, to 
obtain the modulus with respect to the translations of the (M-l) second images. Step 
15 S16 is also to select, based on a criteria, one from the second images whose related 
modulus are the same. The selected second images together with the rest of the 
second images, whose related modulus are not the same, are referred to being as N 
third images where N is equal to or less than (M-l). 

Afterward, step SI 8 is performed to down-sample, according to the respective 
20 translation between each of the N third images and the interpolated prototype image, 
the interpolated prototype image N times to produce N fourth images. Each of the N 
fourth images corresponds to one of the N third images. 

After step SI 8, step S20 is performed to calculate the difference between each 
of the N third images and the corresponding fourth image thereof. 

25 Then, step S22 is performed to adjust the values of the pixels of the interpolated 

prototype image according to an average of the differences calculated in Step 20. 
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Finally, step S24 is performed to determine if the values of the pixels of the 
interpolated prototype image converge to a satisfactory result. If NO in step S24, 
steps SI 8 through S22 are repeated. In the result, the interpolated prototype image, 
whose values of pixels converge to the satisfactory result, is referred to being as the 
enhanced-resolution image. 

In a preferred embodiment, step S 1 8 is implemented according to the following 
formula: 

gl m) =(T k (f w )^h)±s > 
wherein g<"> is the kth forth image at the nth down-sampling process, y<») is 

the interpolated prototype image been, adjusted n times, h is the blurring operator 
defined by a point-spread-function, f is the transformation operator with respect to 

the translation between the kth third image and the interpolated prototype image, Is is 
the down-sampling operator. 

In a preferred embodiment, Step S 1 8 and Step S22 are implemented according 
to the following formula: 

/ (n+i> = / (n) +^ix , (((s* -s* (n) ) t *)./». 

wherein is the interpolated prototype image to be adjusted at («+l) time, 

K is the number of the third images been calculated, g k is the kxh third image, p is 
the de-blurring operator, t~ x is the inverse transformation operator with respect to 

the translation between the kth third image and the interpolated prototype image, ]s is 
the up-sampling operator. 

It is noted that the initial guess phase will significantly affect the of the 
computer system as the iterative method is used to reconstruct an enhanced- 
resolution image. That is, if a better initial guess is applied, great amount of 
computation time will be saved. Because the initial guess is performed merely once 
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Bi, Ci and Di, as i=2, 3, and 4. Finally the value of pixel P is computed with the one- 
dimension interpolation in the vertical direction as demonstrated in FIG. 2A. As 
shown FIG. 2B, the third order interpolation way with one-dimension interpolation is 
used to determine the value of pixel P according to the known values of pixels A\ B\ 
5 C'andD'. 

In the same way, in two-dimension interpolation algorithm, the coefficients of 
f n (x) regarding those interpolation ways of other orders can also be solved. It has 
been observed that the interpolation ways of different orders result in different initial- 
guess images and different convergence rates of image quality as the number of 

10 iteration grows. By selecting the most appropriate order of interpolation, the best 
results of iterative method will be achieved, since the initial guess phase has a great 
influence on the performance of image registration and on the necessary number of 
iterations to achieve the peak result of image. In an experiment, the performances 
about these interpolation ways of different orders are evaluated by Peak Signal-to- 

15 Noise Ratio (PSNR) between the original image and reconstructed images. The 
experimental results are shown in FIG. 3. As shown in FIG. 3, in most situations, the 
third order interpolation way ranks the best choice of initial guess if both complexity 
and reconstructed image quality are concerned. The method according to the 
invention prefers to use only the first order interpolation, the third order interpolation, 

20 or the fifth order interpolation. 

In a preferred embodiment, while the (M-l) second images are the low- 
resolution image sequences associated with a moving object, the translation ( i j ; ), in 
step S14, between the ith second image and the interpolated prototype image is 
calculated according to the following steps of: 

25 (a) assigning at least one interesting point Pi(x,y) on the z'th second image, each 
interesting point corresponding to a local window (w) and a point (u,v) on the 
interpolated prototype image; 
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(b) calculating an absolute difference ( iAD t (x,y\ w, v) ) between each interesting 
point Pi(x,y) and the corresponding point (w,v) thereof according to the 
following formula: 

LAD; (x,y;u,v) = ^ |/, (x + m, y + n) - I Q (u + m, v + «)| 

(m,«)ew ; 

5 (c) calculating a minimum ( ZJ? . (x, ) of the absolute difference LAD^x, y;u,v) 
calculated in step (b) according to the following formula: 

LR ! (x,y) = 2iYg > minLAD i (x,y;u,v) ; and 

(«,v) 

(d) calculating the translation ( it. ) with respect to each interesting point P£xy) 
according to the following formula: 

10 LT t (x,y) = LR t {x,y) - (x,y) • magnification factor . 

In this preferred embodiment, in order to get more accurate image registration 
and then reconstruct the enhanced-resolution image of a moving object, the 
interesting points are selected under the following constraints. 

a. The gradient at the interesting point should be larger than a threshold. 

15 b. For each interesting point on a low-resolution image, the complexity 

corresponding to the corresponding point on the simulated enhanced-resolution 
image should be higher than the complexity of the peripheral points. 

c. The translation between each interesting point and the corresponding point 
should not be zero. 

20 The technology mentioned above is also called the local matching technology as 

shown in FIG. 4A. 
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In another preferred embodiment, while the (M-l) second images are the low- 
resolution image sequences associated with a stationary object, the translation {lt. ), 
in step SI 4, between the ith second image and the interpolated prototype image is 
calculated according to the following steps of: 

5 (a) calculating an absolute difference ( GAD(u, v) ) with respect to the rth second 
image according to the following formula: 

GAD g (u,v)= ^\iAx>y)-h(u+ x > v+ yi 

wherein the point (w,v) is a corresponding position point of the interpolated prototype 
image which is corresponding to a start point of the ith second image corresponds to; 

1 0 (b) calculating a minimum ( GR(i) ) of the absolute difference ( GAD(u, v) ) calculated 
in step (a) according to the following formula: 

GR(i) = arg min GAD, (u, v) , 

(«,v) 

and considering GR(i) as LT i • 

The above-mentioned technology is also called the global matching technology 
15 as shown in FIG. 4B. 

In step SI 6, the objective of the automatic image selection is to select a minimal 
number of useful low-resolution images for the proceeding image processes such as 
image registration. This way can reduce unnecessary calculation to accelerate the 
reconstruction of the enhanced-resolution image. The principles of selecting useful 
20 low-resolution images from the original low-resolution images, associated with a 
stationary scene or a moving object, are different. The different principles will be 
illustrates as follows. 

In an embodiment, while the (M-l) second images are the low-resolution image 
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sequences associated with a moving object, the principle of selecting useful low- I 
resolution images in step SI 6 is that for two second images i 9 j having the same 
modulus and ( W ., V( )=7; > (u J ,v J ) = T J > the second image / is selected if 
GADXu^v^KGADjiUj^Vj) . j 

5 In an embodiment, while the (M-l) second images are the low-resolution image 

sequences associated with a stationary scene, the principle of selecting low-resolution | 
images in step S16 is that for two second images i and j having the same modulus, 
and ( w . ?V/ ) = 7^ and (u J9 v.) = Tj> the second image i is selected if <j] <<x y 2 . The 
variance of {LT, (x,y) \ (x,y) e /,} is defined as cr, 2 -cr 2 +o" 2 • Symbol cr 2 and 
1 0 <J 2 yi are the variances of the translation values along x and y axes respectively. 

| 

Referring to Table 1, Table 1 indicates the performances of a system applying 
the method according to the invention. This system is equipped with a Intel Pentium I 
III CPU and 128MB RAMs. In this case, five 62x62 low-resolution images are used ! 
to reconstruct an enhanced-resolution image based on a magnification factor of 3. In 
1 5 Table 1 , the results of the system executing automatic image selection or not are also f 
listed. .-; 



Table 1 





Computation 
Time (seconds) 


PSNR (db) 


Local 
Matching 
Technique 


With Image 
Selection 


155.4 


26.78 


Without 
Image 
Selection 


582.4 


26.66 


Global 
Matching 
Technique 


With Image 
Selection 


75.8 


26.78 


Without 
Image 
Selection 


496.2 


26.66 



According to the results in Table 1, it is evident that the system executing the 
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automatic image selection proposed by the invention can reduce the computation 
time significantly. Some cases of image reconstruction using the method of the 
invention will described as follows to further prove the performance of system 
applying the method of the invention. 

Referring to FIGS. 5A through FIG. 5D, a first case is illustrated. FIG. 5A 
shows one of low-resolution images. FIG. 5B shows the result after the initial guess 
phase. FIG. 5C shows the reconstructed image after 100 iterations. FIG. 5D shows 
the reconstructed enhanced-resolution image with high-pass filtering process to 
sharpen edge thereof. 

Referring to FIGS. 6 A through FIG. 6D, a second case is illustrated. In the 
second case, an original image is taken as a stationary scene, and is down-sampled by 
a simulated camera to produce a plurality of low-resolution image sequences. Using 
the simulated camera, these points on the original image, which the beginning points 
of the low-resolution images correspond to, are different, i.e., the simulated camera 
moves when capturing the low-resolution images. Then, the low-resolution images 
are used to reconstruct an enhanced-resolution image with a magnification factor of 4, 
in accordance with the method of the invention. FIG. 6A shows one of low- 
resolution images in the second case. FIG. 6B shows the result after the initial guess 
phase. FIG. 6C shows the reconstructed image after 100 iterations. FIG. 6D shows 
the reconstructed enhanced image with a post image process. As shown in FIG. 7, 
the PSNR between the original image and the reconstructed image in the second case 
converges rapidly as the number of iterations grows. By the result of FIG. 7, it is 
evident that the system applying the method of the invention provides well 
performance. 

Referring to FIGS. 8 A through FIG. 8D, a third case is illustrated. In the third 
case, a plurality of low-resolution images, associated with a moving object relative to 
a stationary background, are used to reconstruct an enhanced-resolution image based 
on a magnification factor of 2. FIG. 8A shows one of low-resolution image 
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sequences. FIG. 8B shows the result after the initial guess phase. FIG. 8C shows the 
reconstructed image after 100 iterations. FIG. 8D shows the reconstructed enhanced 
image with a post image process. From FIG. 8A to FIG. 8D, apparently, as the 
number of iteration increases, the object becomes more distinct while the background 
5 becomes blurry and the edges of words are more discernible. 

To sum up the above embodiments and practical cases, the invention provides a 
method that reconstructs enhanced-resolution images with improved enhanced- 
resolution algorithms, and employs the suggested initial interpolation, automatic 
image selection and robust image registration. According to the method of the 

10 invention, the reconstruction of enhanced-resolution image can be quickly achieved 
not only by low-resolution image sequences associated with a stationary scene but 
also by low-resolution image sequences associated with a moving object. The 
reconstructed enhanced-resolution images can further perform the post process for 
enhancing the image quality. Obviously, the method of the invention has the novelty 

1 5 and non-obviousness and satisfies the guideline of patent application. 

With the example and explanations above, the features and spirits of the 
invention will be hopefully well described. Those skilled in the art will readily 
observe that numerous modifications and alterations of the device may be made 
while retaining the teaching of the invention. Accordingly, the above disclosure 
20 should be construed as limited only by the metes and bounds of the appended claims. 
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